Program, information processing method, and information processing device

ABSTRACT

There is provided a program causing a computer to execute processing including: acquiring an endoscopic image of a subject from an endoscope; acquiring a three-dimensional medical image obtained by capturing an image of an internal body portion of the subject by means of X-ray CT, X-ray cone beam CT, MRI-CT, or an ultrasonic diagnosis apparatus configured to capture a three-dimensional image of an internal body portion of the subject; generating a virtual endoscopic image reconfigured from the three-dimensional medical image, based on the acquired endoscopic image; calculating distance image information in the endoscopic image based on the virtual endoscopic image and the endoscopic image; and outputting operation assistance information on an operation of the endoscope based on the distance image information and the three-dimensional medical image.

TECHNICAL FIELD

The present technology relates to a program, an information processingmethod, an information processing apparatus, and a diagnosis assistancesystem.

The present application claims priority based on Japanese PatentApplication No. 2020-056712 filed on Mar. 26, 2020,the entire contentsof which are incorporated herein by reference.

BACKGROUND ART

In a tumor examination of a patient, an endoscope is inserted into atubular organ portion such as the trachea and the bronchus, the uppergastrointestinal tract, the pancreas, the biliary tract, or theintestinal tract, and the examination is mostly performed based on animage from the inserted endoscope. However, in two-dimensional imageinformation of an endoscopic image, a distance to each pixel is notknown, a geometric distortion of the image occurs, and an error in imagemeasurement is large. As a result, it is difficult to provide imagediagnosis support information by using only an endoscopic image. On theother hand, a virtual endoscope disclosed in Patent Literature 1provides a virtual endoscopic image using data of an X-ray computedtomography (CT) image. The virtual endoscopic image is created from athree-dimensional X-ray CT image.

CITATION LIST Patent literature

Patent Literature 1: IP 2002-238887 A

SUMMARY OF INVENTION Technical Problem

However, the virtual endoscope disclosed in Patent Literature 1 merelydisplays an X-ray CT reconfigured cross-section image (virtualendoscopic image), and assisting diagnosis by providing information onan operation of the endoscope based on the endoscopic image and thevirtual endoscopic image is not taken into account.

In one aspect, an object is to provide a program and the like thatperform efficient diagnosis assistance by providing information on anoperation of an endoscope.

Solution to Problem

According to an aspect of the present disclosure, there is provided aprogram causing a computer to execute processing including: acquiring anendoscopic image of a subject from an endoscope; acquiring athree-dimensional medical image obtained by capturing an image of aninternal body portion of the subject by means of X-ray CT, X-ray conebeam CT, MRI-CT or an ultrasonic diagnosis apparatus configured tocapture a three-dimensional image of an internal body portion of thesubject; generating a virtual endoscopic image reconfigured from thethree-dimensional medical image, based on the acquired endoscopic image:calculating distance image information in the endoscopic image based onthe virtual endoscopic image and the endoscopic image; and outputtingoperation assistance information on an operation of the endoscope basedon the distance image information and the three-dimensional medicalimage.

According to another aspect of the present disclosure, there is providedan information processing method causing a computer to executeprocessing including: acquiring an endoscopic image of a subject from anendoscope; acquiring a three-dimensional medical image obtained bycapturing an image of an internal body portion of the subject by meansof X-ray CT, X-ray cone beam CT, MRI-CT, or an ultrasonic diagnosisapparatus configured to capture a three-dimensional image of an internalbody portion of the subject; generating a virtual endoscopic imagereconfigured from the three-dimensional medical image, based on theacquired endoscopic image; calculating distance image information in theendoscopic image based on the virtual endoscopic image and theendoscopic image; and outputting operation assistance information on anoperation of the endoscope based on the distance image information andthe three-dimensional medical image.

According to still another aspect of the present disclosure, there isprovided an information processing apparatus including: an endoscopicimage acquisition unit that acquires an endoscopic image of a subjectfrom an endoscope; a three-dimensional medical image acquisition unitthat acquires a three-dimensional medical image obtained by capturing animage of an internal body portion of the subject by means of X-ray CT,X-ray cone beam CT, MRI-CT, or an ultrasonic diagnosis apparatusconfigured to capture a three-dimensional image of an internal bodyportion of the subject; a generation unit that generates a virtualendoscopic image reconfigured from the three-dimensional medical image,based on the acquired endoscopic image; a calculation unit thatcalculates distance image information in the endoscopic image based onthe virtual endoscopic image and the endoscopic image; and an outputunit that outputs operation assistance information on an operation ofthe endoscope based on the distance image information and thethree-dimensional medical image.

According to still another aspect of the present disclosure, there isprovided a diagnosis assistance system including: an endoscope; anautomatic operation mechanism that performs an automatic operation ofthe endoscope; and an endoscope processor that acquires an endoscopicimage of a subject from the endoscope, in which the endoscope processorincludes a three-dimensional medical image acquisition unit thatacquires a three-dimensional medical image obtained by capturing animage of an internal both portion of the subject by means of X-ray CT,X-ray cone beam CT, WU-CT, or an ultrasonic diagnosis apparatusconfigured to capture a three-dimensional image of an internal bodyportion of the subject, a generation unit that generates a virtualendoscopic image reconfigured from the three-dimensional medical image,based on the acquired endoscopic image, a calculation unit thatcalculates distance image information in the endoscopic image based onthe virtual endoscopic image and the endoscopic image, and an outputunit that outputs operation assistance information on an operation ofthe endoscope to the automatic operation mechanism based on the distanceimage information and the three-dimensional medical image, and in whichthe automatic operation mechanism performs an automatic operation of theendoscope according to the operation assistance information output fromthe output unit.

Advantageous Effects of invention

According to the present disclosure, it is possible to provide a programand the like that perform efficient diagnosis assistance by providinginformation on an operation of the endoscope.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram illustrating an outline of a diagnosisassistance system according to a first embodiment.

FIG. 2 is a block diagram illustrating a configuration example of anendoscope apparatus included in the diagnosis assistance system.

FIG. 3 is a block diagram illustrating a configuration example of aninformation processing apparatus included in the diagnosis assistancesystem.

FIG. 4 is an explanatory diagram exemplifying a data layout of anendoscopic image DB.

FIG. 5 is an explanatory diagram for explaining processing of outputtingoperation assistance information using an operation information learningmodel.

FIG. 6 is an explanatory diagram for explaining processing of outputtinga matching degree with an endoscopic image using a matching degreelearning model.

FIG. 7 is a functional block diagram exemplifying functional unitsincluded in a control unit of the information processing apparatus.

FIG. 8 is an explanatory diagram illustrating an insertion distance (avalue of an S coordinate) of an endoscope.

FIG. 9 is an explanatory diagram on a relationship between an endoscopicimage and a three-dimensional medical image.

FIG. 10 is a flowchart illustrating an example of a processing procedureperformed by the control unit of the information processing apparatus.

FIG. 11 is an explanatory diagram illustrating an aspect of anintegrated image display screen.

FIG. 12 is a functional block diagram exemplifying functional unitsincluded in a control unit of an information processing apparatusaccording to a second embodiment (bending history).

FIG. 13 is a flowchart illustrating an example of a processing procedureperformed by the control unit of the information processing apparatus.

FIG. 14 is a schematic diagram illustrating an outline of a diagnosisassistance system according to a third embodiment (automatic operationmechanism).

FIG. 15 is a functional block diagram exemplifying functional unitsincluded in the control unit of the information processing apparatus.

DESCRIPTION OF EMBODIMENTS First Embodiment

Hereinafter, the present invention will be specifically described withreference to the drawings illustrating embodiments of the presentinvention. FIG. 1 is a schematic diagram illustrating an outline of adiagnosis assistance system S according to a first embodiment. Thediagnosis assistance system S includes an endoscope apparatus 10 and aninformation processing apparatus 6 communicatively connected to theendoscope apparatus 10.

The endoscope apparatus 10 transmits an image (captured image) capturedby an image sensor of an endoscope 40 to an endoscope processor 20, andthe endoscope processor 20 performs various types of image processingsuch as gamma correction, white balance correction, and shadingcorrection. Thereby, an endoscopic image easily observed by an operatoris generated. The endoscope apparatus 10 outputs (transmits) thegenerated endoscopic image to the information processing apparatus 6. Ina case where the information processing apparatus 6 acquires theendoscopic image transmitted from the endoscope apparatus 10, theinformation processing apparatus 6 performs various types of informationprocessing based on these endoscopic images and outputs information ondiagnosis assistance.

The endoscope apparatus 10 includes the endoscope processor 20, theendoscope 40, and a display device 50. The display device 50 is, forexample, a liquid crystal display device or an organic electroluminescence (EL) display device.

The display device 50 is provided on an upper stage of a storage shelf16 with casters. The endoscope processor 20 is accommodated in a middlestage of the storage shelf 16. The storage shelf 16 is disposed in thevicinity of a bed for an endoscopic examination (not illustrated). Thestorage shelf 16 includes a drawer type shelf on which a keyboard 15connected to the endoscope processor 20 is provided.

The endoscope processor 20 has a substantially rectangularparallelepiped shape and is provided with a touch panel 25 on onesurface. A reading unit 28 is disposed at a bottom portion of the touchpanel 25. The reading unit 28 is a connection interface for performingreading and writing on a portable recording medium such as a universalserial bus (USB) connector, a secure digital (SD) card slot, a compactdisc read only memory (CD-ROM) drive, or the like.

The endoscope 40 includes an insertion portion 44, an operation unit 43,a universal cord 49, and a scope connector 48. The operation unit 43 isprovided with a control button 431. The insertion portion 44 is long andhas one end connected to the operation unit 43 via a bend preventingportion 45. The insertion portion 44 includes a soft portion 441, abending portion 442, and a distal end portion 443 in order from a sideof the operation unit 43. The bending portion 442 is bent according toan operation of a bending knob 433. Physical detection devices such as athree-axis acceleration sensor, a gyro sensor, a geomagnetic sensor, amagnetic coil sensor, and an endoscope-insertion-type observation device(colonoscope navigation) may be provided on the insertion portion 44. Ina case where the endoscope 40 is inserted into a body of a subject,detection results from these physical detection devices may be acquired.

The universal cord 49 is long and has a first end connected to theoperation unit 43 and a second end connected to the scope connector 48.The universal cord 49 is soft. The scope connector 48 has asubstantially rectangular parallelepiped shape. The scope connector 48is provided with an air supply/water supply port 36 (refer to FIG. 2)for connecting an air supply/water supply tube.

FIG. 2 is a block diagram illustrating a configuration example of theendoscope apparatus 10 included in the diagnosis assistance system S. Acontrol unit 21 is an arithmetic control device that executes a programaccording to the present embodiment. One or a plurality of centralprocessing units (CPUs), graphics processing units (GPUs), multi-coreCPUs, or the like is used for the control unit 21. The control unit 21is connected to each hardware unit of the endoscope processor 20 via abus.

A main storage device 22 is, for example, a storage device such as astatic random access memory (SRAM), a dynamic random access memory(DRAM), or a flash memory. The main storage device 22 temporarily storesinformation required in the middle of processing performed by thecontrol unit 21 and a program being executed by the control unit 21. Anauxiliary storage device 23 is, for example, a storage device such as anSRAM, a flash memory, or a hard disk and is a storage device having acapacity larger than that of the main storage device 22. In theauxiliary storage device 23, for example, the acquired captured imageand the generated endoscopic image may be stored as intermediate data.

A communication unit 24 is a communication module or a communicationinterface for performing communication with the information processingapparatus 6 via a network in a wired or wireless manner and is, forexample, a narrow-area wireless communication module such as Wi-Fi(registered trademark) or Bluetooth (registered trademark) or awide-area wireless communication module such as 4G or Long TermEvolution (LTE). The touch panel 25 includes a display unit such as aliquid crystal display panel and an input unit layered on the displayunit. The communication unit 24 may perform communication with a CTapparatus, an MRI apparatus (refer to FIG. 5), an ultrasonic diagnosisapparatus, or a storage device (not illustrated) that stores data outputfrom these apparatuses.

A display device I/F 26 is an interface for connecting the endoscopeprocessor 20 and the display device 50. An input device I/F 27 is aninterface for connecting the endoscope processor 20 and an input devicesuch as the keyboard 15.

A light source 33 is a high-luminance white light source such as a whiteLED or a xenon lamp. The light source 33 is connected to the bus via adriver (not illustrated). In the light source 33, turning on, fumingoff, and a change of luminance are controlled by the control unit 21.Illumination light emitted from the light source 33 is incident on anoptical connector 312. The optical connector 312 engages with the scopeconnector 48 to supply the illumination light to the endoscope 40.

A pump 34 generates a pressure for the air supply and water supplyfunction of the endoscope 40. The pump 34 is connected to the bus via adriver (not illustrated). In the pump 34, turning on, turning off, and achange of the pressure are controlled by the control unit 21. The pump34 is connected to the air supply/water supply port 36 provided in thescope connector 48 via a water supply tank 35.

An outline of functions of the endoscope 40 connected to the endoscopeprocessor 20 will be described. A fiber bundle, a cable bundle, an airsupply tube, a water supply tube, and the like are inserted inside thescope connector 48, the universal cord 49, the operation unit 43, andthe insertion portion 44. The illumination light emitted from the lightsource 33 is emitted from an illumination window provided at the distalend portion 443 via the optical connector 312 and the fiber bundle. Theimage sensor provided at the distal end portion 443 captures an image ofa range illuminated by the illumination light. The captured image istransmitted from the image sensor to the endoscope processor 20 via thecable bundle and an electrical connector 311.

The control unit 21 of the endoscope processor 20 functions as an imageprocessing unit 211 by executing a program stored in the main storagedevice 22. The image processing unit 211 performs various types of imageprocessing such as gamma correction, white balance correction, andshading correction on the image (captured image) output from theendoscope 40 and outputs the image as the endoscopic image.

FIG. 3 is a block diagram illustrating a configuration example of theinformation processing apparatus 6 included in the diagnosis assistancesystem S. The information processing apparatus 6 includes a control unit62, a communication unit 61, a storage unit 63, and an input/output I/F64. The information processing apparatus 6 is, for example, a serverapparatus, a personal computer, or the like. The server apparatusincludes not only a single server apparatus but also a cloud serverapparatus or a virtual server apparatus including a plurality ofcomputers. The information processing apparatus 6 may be provided as acloud server located on an external network accessible from theendoscope processor 20.

The control unit 62 includes one or a plurality of arithmetic processingdevices having a time counting function, such as central processingunits (CPUs), micro-processing units (MPUs), and graphics processingunits (CPUs), and performs various types of information processing,control processing, and the like related to the information processingapparatus 6 by reading and executing a program P stored in the storageunit 63. Alternatively, the control unit 62 may include a quantumcomputer chip, and the information processing apparatus 6 may be aquantum computer.

The storage unit 63 includes a volatile storage area such as a staticrandom access memory (SRAM), a dynamic random access memory (DRAM), or aflash memory and a nonvolatile storage area such as an EEPROM or a harddisk. The storage unit 63 stores in advance the program P and data to bereferred to at the time of processing. The program P stored in thestorage unit 63 may he a program P which is read from a. recordingmedium 632 readable by the information processing apparatus 6. Inaddition, the program P may be a program which is downloaded from anexternal computer (not illustrated) connected to a communication network(not illustrated) and is stored in the storage unit 63. The storage unit63 stores an entity file (instance file of a neural network (NN))constituting each of a plurality of learning models (91 and 92) to bedescribed later. These entity files may be configured as a part of theprogram P. Further, the storage unit 63 may store an endoscopic imagedatabase (DB) 631 to be described later.

The communication unit 61 is a communication module or a communicationinterface for performing communication with the endoscope apparatus 10in a wired or wireless manner and is, for example, a narrow-areawireless communication module such as Wi-Fi (registered trademark) orBluetooth (registered trademark) or a wide-area wireless communicationmodule such as 4G or LTE. The communication unit 61 may performcommunication with a CT apparatus, an MR1 apparatus (refer to FIG. 5),an ultrasonic diagnosis apparatus, or a storage device (not illustrated)that stores data output from these apparatuses.

The input/output I/F 64 is a communication interface conforming, forexample, to a communication standard such as USB or DSUB and is acommunication interface for performing serial communication with anexternal apparatus connected to the input/output I/F 64. For example, adisplay unit 7 such as a display and an input unit 8 such as a keyboardare connected to the input/output I/F 64, and the control unit 62outputs, to the display unit 7, a result of information processingperformed based on an execution command or an event input from the inputunit 8.

FIG. 4 is an explanatory diagram exempl4ing a data layout of theendoscopic image DB 631. The endoscopic image DB 631 is stored in thestorage unit 63 of the information processing apparatus 6, and isconfigured by database management software such as a relational databasemanagement system (RDBMS) implemented in the information processingapparatus 6. Alternatively, the endoscopic image DB 631 may be stored ina predetermined storage area accessible from the information processingapparatus 6, such as a storage device which is communicatively connectedto the information processing apparatus 6. Alternatively, the endoscopicimage DB 631 may be stored in the main storage device 22 of theendoscope apparatus 10. That is, the predetermined storage area includesthe storage unit 63 of the information processing apparatus 6, the mainstorage device 22 of the endoscope apparatus 10, and a storage deviceaccessible from the information processing apparatus 6 or the endoscopeapparatus 10. The information processing apparatus 6 may acquire theendoscopic image which is output by the endoscope processor 20, anexamination dale and time, and attribute information of the subject, andregister the acquired endoscopic image, the acquired examination dateand time, and the acquired attribute information of the subject in theendoscopic image DB 631. Alternatively, the endoscopic image which isdirectly output from the endoscope processor 20, the examination dateand time, and the attribute information of the subject may be directlyregistered in the endoscopic image DB 631.

The endoscopic image DB 631 includes, for example, a subject mastertable and an image table, and the subject master table and the imagetable are set to be associated with each other by a subject ID that is acommon item (metadata) included in both the tables.

The subject master table includes, for example, the subject ID, agender; a date of birth, and an age as management items (metadata). Inthe item (field) of the subject ID, ID information is stored in order touniquely specify the subject who has an endoscopic examination. In theitems (fields) of the gender and the date of birth, biologicalattributes including the gender and the date of birth corresponding tothe subject ID are stored. In the item (field) of the age, the age at acurrent time calculated based on the date of birth is stored. The genderand the age are managed as biological information of the subject by thesubject master table.

The image table includes, for example, as management items (metadata), asubject ID, an examination date and time, an endoscopic image, a framenumber, an S coordinate (insertion distance), a three-dimensionalmedical image, a viewpoint position, a viewpoint direction, and avirtual endoscopic image.

In the item (field) of the subject ID, a value of the subject ID that isassociated with the biological attribute of the subject managed in thesubject master table is stored. In the item (field) of the examinationdate and time, a date and time when the subject corresponding to thesubject ID has the endoscopic examination is stored. In the item (field)of the endoscopic image, the endoscopic image corresponding to thesubject ID is stored as object data. The endoscopic image may be a stillimage in, tier example, a jpeg format with one frame or a moving imagein, for example, an avi format with several frames. In the item (field)of the endoscopic image, information indicating a storage location (filepath) of the endoscopic image stored as a file may be stored.

In a case where the endoscopic image is a moving image, in the item(field) of the frame number, the frame number of the moving image isstored. Even in a case where the endoscopic image is a moving image, bystoring the frame number of the moving image, the moving image can behandled in the same manner as a still image, and can be associated withposition information (a coordinate in an internal body coordinatesystem) of a three-dimensional medical image or a virtual endoscopicimage to be described later.

In the item (field) of the S coordinate (insertion distance), theinsertion distance of the endoscope 40 at the time of capturing theendoscopic image to be stored in the same record is stored as a value ofthe S coordinate. Calculation of the insertion distance (S coordinate)be described later.

In the item (field) of the three-dimensional medical image, for example,a three-dimensional medical image in a digital imaging andcommunications in medicine (DICOM) format is stored as object data, thethree-dimensional medical image being generated based on data outputfrom means configured to capture a three-dimensional image of aninternal portion of the body, such as a CT apparatus (X-ray CT, X-raycone beam CT), an MRI apparatus (MRI-CT), or an ultrasonic diagnosisapparatus. Alternatively, in the item (field) of the three-dimensionalmedical image, information indicating a storage location (file path) ofthe three-dimensional medical image stored as a file may he stored.

In the item (field) of the viewpoint position, a coordinate of theendoscope 40 in the body at the time of capturing the endoscopic image,that is, a coordinate in a coordinate system of the three-dimensionalmedical image is stored. Calculation of the viewpoint position will bedescribed later.

In the item (field) of the viewpoint direction, a direction of theendoscope 40 at the time of capturing the endoscopic image, that is, arotation angle in a coordinate system (a coordinate in the internal bodycoordinate system) of the three-dimensional medical image is stored.Calculation of the viewpoint direction will be described later.

In the item (field) of the virtual endoscopic image, the virtualendoscopic image generated from the three-dimensional medical image isstored as object data. In the item (field) of the virtual endoscopicimage, information indicating a storage location (file path) of thevirtual endoscopic image stored as a file may be stored. The virtualendoscopic image is generated from the three-dimensional medical imagein order to perform matching processing with the endoscopic image. Forexample, a virtual endoscopic image having a highest matching degreewith the endoscopic image is registered in the same record as theendoscopic image. Generation of the virtual endoscopic image will bedescribed later.

FIG. 5 is an explanatory diagram for explaining processing of outputtingoperation assistance information using an operation information learningmodel 91. The information processing apparatus 6 configures (generates)a neural network (operation information learning model 91) that receivesdistance image information and a three-dimensional medical image andoutputs operation assistance information including an insertiondirection of the endoscope 40, by performing learning based on trainingdata. by setting, as inquiry data, distance image information to bedescribed later and body cavity information included in athree-dimensional medical image and setting, as answer data, operationassistance information including at least one of an insertion direction,an insertion amount, and an insertion speed of the endoscope 40, and atarget point coordinate indicating an insertion destination of theendoscope 40.

It is assumed that the operation information learning model 91 learnedusing the training data is used as a program module which is a part ofartificial intelligence software. The operation information learningmodel 91 is used in the information processing apparatus 6 including thecontrol unit 62 (CPU or the like) and the storage unit 63 as describedabove, and is executed. by the information processing apparatus 6 havingarithmetic processing capability. Thereby, a neural network system isconfigured. That is, the control unit 62 of the information processingapparatus 6 operates to perform an arithmetic operation of extractingfeature amounts of the distance image information and thethree-dimensional medical image, which are input into an input layer,according to a command. from the operation information learning model 91stored in the storage unit 63 and output the operation assistanceinformation including the insertion direction of the endoscope 40 froman output layer.

The input layer includes a plurality of neurons that receive thedistance image information and the body cavity information included inthe three-dimensional medical image and transfer the input distanceimage information and the input body cavity information included in thethree-dimensional medical image to an intermediate layer. Althoughdescribed in detail later, the distance image information is informationcalculated based on a virtual endoscopic image corresponding to theacquired endoscopic image, and is information on a distance betweenpixels in the virtual endoscopic image. Since the endoscopic image andthe corresponding virtual endoscopic image include, as an imagingregion, a region of the same internal body part, the distance imageinformation corresponds to information on the distance between pixels inthe endoscopic image. The distance between pixels means a distance inthe coordinate system (internal body coordinate system) of thethree-dimensional medical image, and is, for example, a distance inconsideration of a depth in two internal body parts included in thevirtual endoscopic image. Further, information on the viewpoint positionand the direction of the endoscope 40 may be reflected to the distanceimage information which is input to the operation information learningmodel 91. The body cavity information included in the three-dimensionalmedical image is curved surface data indicating a shape (a shape of aninner wall of an organ) of an internal organ into which the endoscope 40is inserted in a three-dimensional region including the imaging regionof the virtual endoscopic image from which the distance imageinformation is calculated. The curved surface data may be representedby, for example, a polynomial approximation expression or a set ofpoints.

The intermediate layer has, for example, a single phase or a multilayerstructure including one or a plurality of fully-connected layers, andeach of a plurality of neurons included in the fully-connected layersoutputs information indicating activation or deactivation based on theinput distance image information and a value of the input body cavityinformation included in the three-dimensional medical image. Theinformation processing apparatus 6 optimizes parameters used forarithmetic processing in the intermediate layer by using, for example,backpropagation or the like.

The output layer includes one or a plurality of neurons that outputoperation assistance information including the insertion direction ofthe endoscope 40, and outputs the operation assistance information basedon the information indicating activation or deactivation of each neuronthat is output from the intermediate layer. The operation assistanceinformation including the insertion direction of the endoscope 40 may bedisplayed, for example, in a vector format in the coordinate system(internal body coordinate system) of the three-dimensional medicalimage, the vector format including a plurality of coordinate values anda plurality of rotation angles through which the distal end portion ofthe endoscope 40 sequentially passes in the insertion direction.Further, the operation assistance information may include a speedcomponent with respect to a movement amount between values of adjacentcoordinates through which the endoscope 40 sequentially passes.

Pieces of the inquiry data of the distance image information and thethree-dimensional medical image (intermediate data) that are used as thetraining data and the operation assistance information including theinsertion direction of the endoscope 40 and correlated with these piecesof the information are stored in a large amount as result data ofexaminations performed using the endoscope 40 in each medicalinstitution. By using these pieces of the result data, a large amount oftraining data for learning the operation information learning model 91can be generated.

FIG. 6 is an explanatory diagram for explaining processing of outputtinga matching degree with the endoscopic image using a matching degreelearning model 92. The information processing apparatus 6 configures(generates) a neural network (matching degree learning model 92) thatreceives the endoscopic image and the virtual endoscopic image andoutputs information such as a value indicating a matching degree of bothimages, by performing learning based on training data by setting, asinquiry data, the endoscopic image and the virtual endoscopic image andsetting, as answer data, information on a matching degree of bothimages. It is assumed that, similarly to the operation informationlearning model 91, the matching degree learning model 92 learned usingthe training data is used as a program module which is a part ofartificial intelligence software.

The input layer includes a plurality of neurons that receive pixelvalues of the endoscopic image and the virtual endoscopic image andtransfer the input pixel values to an intermediate layer. Theintermediate layer includes a plurality of neurons that extract imagefeature amounts of the endoscopic image and the virtual endoscopic imageand transfer the extracted image feature amounts of both images to anoutput layer. The output layer includes one or a plurality of neuronsthat output information on a matching degree such as a value indicatingthe matching degree of the input endoscopic image and the input virtualendoscopic image, and outputs the information on the matching degreebased on the image feature amounts of both images which are output fromthe intermediate layer.

For example, in a case where the matching degree learning model 92 is aconvolutional neural network (CNN), the intermediate layer has aconfiguration in which a convolution layer that convolves a pixel valueof each pixel which is input from the input layer and a pooling layerthat maps (compresses) the pixel value convolved by the convolutionlayer are alternately connected, and the intermediate layer finallyextracts the feature amounts of the endoscopic image and the virtualendoscopic image while compressing pixel information of the endoscopicimage and pixel information of the virtual endoscopic image. The outputlayer includes, for example, a fully-connected layer and a soft maxlayer. The fully-connected layer calculates a cosine similarityaccording to an inner product of feature amount vectors based on imagefeature amounts of both images, and the soft max layer calculates avalue (established value) indicating a matching degree based on thecosine similarity. Thus, the output layer outputs the value asinformation on the matching degree.

In configuration (generation) of the matching degree learning model 92,for example, by using a repository (learned model) such as DCNNimplemented in a VGG16 model (caffemodel: VGG_ILSVRC_16_layers),transfer learning may be performed by training data based on theendoscopic image and the virtual endoscopic image, and the matchingdegree learning model 92 may be configured. The endoscopic image used asthe training data and the virtual endoscopic image corresponding to theendoscopic image are stored in a large amount as result data ofexaminations performed using the endoscope 40 and the CT apparatus ineach medical institution. By using these pieces of result data, a largeamount of training data for learning the matching degree learning model92 can be generated.

In the present embodiment, the operation information learning model 91and the matching degree learning model 92 are described as a neuralnetwork (NN) such as a CNN. On the other hand, these learning models (91and 92) are not limited to the N, and may be learning models (91 and 92)including another learning algorithm such as a support vector machine(SVM), a Bayesian network, or a regression tree.

FIG. 7 is a functional block diagram exemplifying functional unitsincluded in the control unit of the information processing apparatus.The control unit 21 of the endoscope processor 20 (endoscope apparatus10) executes the program stored in the main storage device 22, therebyfunctioning as the image processing unit 211. The control unit 62 of theinformation processing apparatus 6 functions as an acquisition unit 621,a viewpoint position calculation unit 622, a virtual endoscopic imagegeneration unit 623, a matching degree determination unit 624, adistance image information calculation unit 625, and an operationassistance information output unit 626 by executing the program P storedin the storage unit 63.

The image processing unit 211 of the endoscope processor 20 performsvarious types of image processing such as gamma correction, whitebalance correction, and shading correction on the image (captured image)output from the endoscope, and outputs the image as the endoscopicimage. The image processing unit 211 outputs (transmits) the generatedendoscopic image and the examination date and time based on an imagingtime of the endoscopic image to the information processing apparatus 6.The image processing unit 211 may further output the subject ID which isinput from the keyboard 15 to the information processing apparatus 6.The image processing unit 211 may output, to the information processingapparatus 6, information on the insertion distance (S coordinate) of theendoscope 40 that is output from a sensor disposed in the insertionportion 44 (flexible tube) of the endoscope 40 in order to measure asurrounding environment of the endoscope 40. The image processing unit211 may superimpose the information on the insertion distance of theendoscope 40 acquired from the sensor, for example, on the endoscopicimage and display the superimposed image on the display device.

The sensor for acquiring the S coordinate which is the insertiondistance of the endoscope 40 in the internal body is, for example, atemperature sensor, an optical sensor, a pressure sensor, a wet sensor(electrode), and a humidity sensor. For example, in a case where thesensor is an optical sensor, the optical sensor is disposed inside theinsertion portion 44 (flexible tube). On the other hand, the opticalsensor can receive light even in a case where the insertion portion 44(flexible tube) is inserted into the body. Therefore, it is possible todetermine that a portion where the optical sensor receives more light isoutside the body and a portion where the optical sensor receives lesslight is inside the body. The control unit 21 of the endoscope processor20 can calculate the S coordinate which is an insertion distance(length) of the insertion portion 44 (flexible tube) in the internalbody by specifying the optical sensor at a boundary positioncorresponding to an insertion-allowance part in the body cavity based ona signal obtained by the optical sensor.

In a case of an upper endoscope, a roller encoder is attached to amouthpiece or the like (not illustrated) in contact with the insertionportion 44 (flexible tube), and the roller encoder is rotated by adistance by which the insertion portion 44 (flexible tube) is insertedinto the body. Thus, it is possible to acquire the S coordinate which isthe insertion distance of the endoscope 40 in the internal body. Theroller encoder attached to a mouthpiece or the like is rotated as theinsertion portion 44 (flexible tube) moves forward and backward. Thus,the roller encoder can measure a length between the distal end portion443 of the endoscope 40 inserted into the body and an openingcommunicating with a lumen such as a mouth or a nose, that is, aninsertion distance of the insertion portion 44 (flexible tube). Theroller encoder is electrically connected to the endoscope processor 20,and transmits the measured distance to the endoscope processor 20.Alternatively, an optical encoder or a magnetic encoder may he usedinstead of the roller encoder.

In addition, in a case of a lower endoscope, an object corresponding toa mouthpiece is attached to an anal portion, and an insertion distanceof the endoscope can be measured. In a case where an auxiliary devicefor measuring the insertion distance of the endoscope 40 is attached toan insertion-allowance part in the body cavity that is an entrance ofthe subject, a passing distance of the endoscope 40 is measured. Thus,the S coordinate which is the insertion distance of the endoscope 40 inthe internal body can be acquired. The auxiliary device may be, forexample, a device that measures a distance using a scale of a magneticfield such as a linear scale attached to the insertion portion (flexibletube) 44 and a linear head attached to a mouthpiece, or may be amouthpiece of the endoscope 40 to which a roller is attached. In a casewhere the endoscope 40 is inserted into a nose, an anus, or the like, anauxiliary device provided with a roller similar to the mouthpiece may beused. Further, a chip in which an insertion distance is recorded atregular intervals may be incorporated in the insertion portion (flexibletube) 44 of the endoscope 40. From S coordinate information that isrecorded in the chip and is obtained by a mouthpiece or the like, theendoscope processor 20 can acquire the S coordinate which is theinsertion distance of the endoscope 40 in the internal body.

The acquisition unit 621 acquires the subject ID, the examination dateand time, the endoscopic image, and the S coordinate (insertiondistance) output by the endoscope processor 20. Based on the acquiredsubject ID, the acquisition unit 621 acquires a three-dimensionalmedical image of the subject that is output from means configured tocapture a three-dimensional image of the internal body, such as a CTapparatus, a cone beam CT apparatus, an MRI apparatus, or an ultrasonicdiagnosis apparatus, which is communicatively connected. In a case wherea three-dimensional medical image, which is output from anotherexamination apparatus configured to capture a three-dimensional image ofthe internal body, such as a CT apparatus, a cone beam CT apparatus, anMRI apparatus, or an ultrasonic diagnosis apparatus, is already storedin, fir example, an external server (not illustrated), the informationprocessing apparatus 6 may access the external server, and acquire thethree-dimensional medical image of the subject based on the subject IDoutput from the endoscope processor 20.

The three-dimensional medical image is, for example, an imagerepresented by volume data including tomographic image data, which isoutput from means configured to capture a three-dimensional image of theinternal body, such as a CT apparatus, a cone beam CT apparatus, an MRIapparatus, or an ultrasonic diagnosis apparatus, or is an imagerepresented by volume data which is output from a Multi Slice CTapparatus or an X-ray cone beam CT apparatus using an X-ray flat panel.In a case where an X-ray CT apparatus or a cone beam CT apparatus isused, for example, dual energy imaging may be performed by the X-ray CT,and an image in which a composition (body composition) of each pixel ofa three-dimensional medical image can be identified by the effectivemass number (effective-Z) may be used. In a case where an MRI apparatusis used, an image obtained by adding information on a composition (bodycomposition) of each pixel of a three-dimensional medical image, such asfat or lactic acid, may be used.

The acquisition unit 621 outputs the acquired S coordinate to theviewpoint position calculation unit 622. Based on the acquired Scoordinate, the viewpoint position calculation unit 622 calculates acoordinate (a coordinate in the internal body coordinate system) of thethree-dimensional medical image corresponding to the S coordinate, thatis, a viewpoint position at which the distal end portion 443 of theendoscope 40 is located. at the time when an image is captured by theendoscope 40. FIG. 8 is an explanatory diagram illustrating an insertiondistance (a value of an S coordinate) of the endoscope 40. Asillustrated in FIG. 8, in the three-dimensional medical image, adigestive organ captured by the endoscope 40 is represented by athree-dimensional shape. A space is formed inside an inner wall of thedigestive organ, and the space serves as an insertion path through whichthe endoscope 40 is inserted. The S coordinate which is the insertiondistance of the endoscope 40 corresponds to a place inside the insertionpath (inside the inner wall of the digestive organ) and where a pathlength of the insertion path is substantially equal to the insertiondistance. Thus, the coordinate of the distal end portion 443 of theendoscope 40 located inside the inner wall of the digestive organ can becalculated based on the S coordinate. The viewpoint position calculationunit 622 outputs information on the calculated viewpoint position to thevirtual endoscopic image generation unit 623.

The acquisition unit 621 outputs the acquired three-dimensional medicalimage to the virtual endoscopic image generation unit 623. The virtualendoscopic image generation unit 623 generates a virtual endoscopicimage based on the acquired three-dimensional medical image and theviewpoint position acquired from the viewpoint position calculation unit622. The virtual endoscopic image is an image which is generated(reconfigured) based on the three-dimensional medical image obtained bycapturing a tubular organ such as a trachea and a bronchus or anintestinal tract by X-ray CT, MRI, or X-ray cone beam CT and in whichthe inside of an organ (inside the body cavity) in the three-dimensionalmedical image is represented by a virtual endoscope. For example, CTimaging may be performed in a state where air is introduced into a largeintestine, and a virtual endoscopic image of the large intestine may begenerated (reconfigured) by performing volume rendering on athree-dimensional medical image obtained by the imaging from the insideof the large intestine.

The virtual endoscopic image generation unit 623 extracts voxel data ofthe organ in the subject from the acquired three-dimensional medicalimage. Examples of the organ include a large intestine, a smallintestine, a kidney, a bronchus, a blood vessel, and the like. On theother hand, the organ is not limited thereto, and array be anotherorgan. In the present embodiment, it is assumed that voxel data of thelarge intestine is extracted and acquired. For example, as a method ofextracting a large intestine region, specifically, first, a plurality ofaxial images having a cross-section (axial cross-section) perpendicularto a body axis are reconfigured based on the three-dimensional medicalimage, for each axial image, a boundary between a body surface and theinside of the body is obtained by setting, as a threshold value, anX-ray CT value based on an X-ray absorption coefficient by a knownmethod, and processing of separating an external body region and aninternal body region is performed based on the body surface as areference. For example, binarization processing using an X-ray CT valueis performed on the reconfigured axial image, a contour is extracted bycontour extraction processing, and the inside of the extracted contouris extracted as the internal body (human body) region. Next,binarization processing using a threshold value is performed on theaxial image of the internal body region, and a candidate large-intestineregion in each axial image is extracted. Specifically, since air is in atube of the large intestine, a threshold value (for example, a valueequal to or lower than −600 HU (Hounsfield Unit)) corresponding to a CTvalue of air is set, and binarization processing is performed. As aresult, in each axial image, an air region in the internal body isextracted as a candidate large-intestine region. The virtual endoscopicimage generation unit 623 reconfigures, as a virtual endoscopic image,an image obtained by central projection of projecting, on apredetermined projection plane, voxel data in a plurality of light beamdirections radially extending around a viewpoint vector based on arotation angle that is set as a viewpoint position and a viewpointdirection. As a specific method of the central projection, for example,a known volume rendering method or the like may be used.

For example, the virtual endoscopic image generation unit 623sequentially generates a plurality of candidate virtual endoscopicimages by changing a viewpoint direction, that is, a rotation angle (Θx,Θy, Θz) in a coordinate system of a three-dimensional medical image, forexample, by a predetermined unit amount of 1°, from a viewpoint positioncorresponding to the coordinate of the distal end portion 443 of theendoscope 40 as a starting point. That is, for example, the virtualendoscopic image generation unit 623 may generate the plurality ofvirtual endoscopic images by projecting a three-dimensional shape of theinner wall of the digestive organ by a plurality of rotation angleswhich are set as the viewpoint direction, from the viewpoint positioninside the digestive organ specified in the three-dimensional medicalimage. The virtual endoscopic image generation unit 623 associates theplurality of generated virtual endoscopic images with the viewpointdirection (rotation angle) used when the virtual endoscopic image isgenerated, and outputs the virtual endoscopic image to the matchingdegree determination unit 624.

The acquisition unit 621 outputs the acquired endoscopic image to thematching degree determination unit 624. Based on the acquired endoscopicimage, the plurality of virtual endoscopic images acquired from thevirtual endoscopic image generation unit 623, and the viewpointdirection (rotation angle) used when the virtual endoscopic image isgenerated, the matching degree determination unit 624 specifies thevirtual endoscopic image having a highest matching degree with theacquired endoscopic image and the viewpoint direction (rotation angle)used when the virtual endoscopic image having a highest matching degreeis generated. The matching degree determination unit 624 calculates amatching degree between the endoscopic image and the virtual endoscopicimage by comparing the acquired endoscopic image with each of theplurality of virtual endoscopic images.

The matching degree determination unit 624 includes a matching degreelearning model 92 that outputs information such as a value indicating amatching degree of both images based on the input endoscopic image andthe input virtual endoscopic image. The matching degree determinationunit 624 may input the acquired endoscopic image and the acquiredvirtual endoscopic image to the matching degree learning model 92, andoutput the virtual endoscopic image having a highest value among values(probability values) indicating the matching degrees output by thematching degree learning model 92, as the virtual endoscopic imagecorresponding to the endoscopic image.

Alternatively, the matching degree determination unit 624 is not limitedto the case where the matching degree learning model 92 is included, andfor example, may measure the matching degree by using an indexindicating a correlation between a shade image of the endoscopic imageand a shade image of the virtual endoscopic image. In order toquantitatively confirm the matching degree between the virtualendoscopic image and the endoscopic image, a level of the matchingdegree may be determined by confirming a correlation of shade imageinformation obtained from luminance information. Alternatively, thematching degree determination unit 624 may compare a similarity betweenthe plurality of configured virtual endoscopic images and the endoscopicimage. The comparison of the similarity between the two images isperformed by known image processing, and either pixel data levelmatching or matching of levels of features extracted from the images maybe used. The virtual endoscopic image which is specified as having ahighest matching degree with the endoscopic image by the matching degreedetermination unit 624 and the viewpoint direction (rotation angle) usedto generate the virtual endoscopic image may be registered in theendoscopic image DB. The matching degree determination unit 624 outputsthe virtual endoscopic image which is specified as having the highestmatching degree with the endoscopic image and the viewpoint position andthe viewpoint direction (rotation angle) used to generate the virtualendoscopic image, to the distance image information calculation unit625.

In the present embodiment, the matching degree determination unit 624specifies the virtual endoscopic image having a highest matching degreewith the acquired endoscopic image. On the other hand, the presentinvention is not limited thereto. The matching degree determination unit624 may specify the virtual endoscopic image of which the matchingdegree is equal to or higher than a predetermined value, as a virtualendoscopic image that is substantially identical to the acquiredendoscopic image. By specifying the virtual endoscopic image of whichthe matching degree is equal to or higher than the predetermined value,it is not necessary to compare all the virtual endoscopic imagesgenerated as candidates. Thus, it is possible to reduce a calculationload and a processing time of the information processing apparatus 6.Alternatively, the matching degree determination unit 624 may specify avirtual endoscopic image having a smallest difference (difference index)from the acquired endoscopic image, as a virtual endoscopic image thatis substantially identical to the acquired endoscopic image. Thedifference (difference index) between the endoscopic image and thevirtual endoscopic image corresponds to a reciprocal of the matchingdegree between the endoscopic image and the virtual endoscopic image.Thus, by using an image comparison engine that calculates such adifference (difference index), it is possible to efficiently acquire avirtual endoscopic image that is substantially identical to the acquiredendoscopic image.

In a case where the matching degree is not equal to or higher than thepredetermined value, the matching degree determination unit 624 mayregenerate a plurality of virtual endoscopic images again from aviewpoint position obtained by finely correcting the viewpoint positionacquired from the viewpoint position calculation unit 622. calculate amatching degree between the plurality of regenerated virtual endoscopicimages and the endoscopic image, and specify the virtual endoscopicimage having a highest matching degree.

The distance image information calculation unit 625 calculates distanceimage information based on the virtual endoscopic image acquired fromthe matching degree determination unit 624. The distance imageinformation is information on a distance between pixels in the virtualendoscopic image in the virtual endoscopic image. The distance betweenpixels means a distance in the coordinate system (internal bodycoordinate system) of the three-dimensional medical image, and is, forexample, a distance in consideration of a depth in two internal bodyparts included in the virtual endoscopic image. The virtual endoscopicimage is a two-dimensional image obtained by performing projectiontransformation on the three-dimensional medical image. A certain pointin the virtual endoscopic image corresponds to a point in thethree-dimensional medical image, and these points indicate the sameposition in the internal body part. A certain point in the virtualendoscopic image may be a pixel number (pixel coordinate) that is asmallest unit of an image, or may be, for example, a central portion ofa local region (region including a plurality of pixels) that specifies apredetermined internal body part. By determining two certain points inthis manner, a distance between the two points in the coordinate systemof the three-dimensional medical image can be calculated. That is, thedistance in the distance image information corresponds to a distancebetween two points in the coordinate system of the three-dimensionalmedical image corresponding to two points in the virtual endoscopicimage.

The two points in the three-dimensional medical image are specified fromthe two points in the virtual endoscopic image. Based on coordinatevalues of the specified two points in the three-dimensional medicalimage, a distance and a vector between the two points can be calculated.By reflecting the calculated distance and the calculated vector betweenthe two points in the three-dimensional medical image to the virtualendoscopic image, as a distance and a vector between two points in thevirtual endoscopic image corresponding to the two points, it is possibleto generate a distance image, that is, a virtual endoscopic image(distance image) obtained by reflecting the distance information in thecoordinate system of the three-dimensional medical image to the virtualendoscopic image. The distance image information calculation unit 625may output the virtual endoscopic image (distance image) to which theinformation on the distance between pixels is reflected as the distanceimage information. Further, the distance image information calculationunit 625 may output the virtual endoscopic image to which the viewpointposition and the direction of the endoscope 40 used to generate thevirtual endoscopic image are reflected.

The endoscopic image corresponds to a virtual endoscopic imagereconfigured (generated) from the three-dimensional medical image basedon the position (viewpoint position) and the imaging direction(viewpoint direction) of the endoscope 40 that captures the endoscopicimage. Thus, the distance image information based on the virtualendoscopic image corresponding to the endoscopic image can also beapplied to the endoscopic image. That is, a distance between two pointsin the endoscopic image corresponds to a distance (a distance in thedistance image, a distance in the coordinate system of thethree-dimensional medical image) between two points in the virtualendoscopic image corresponding to the endoscopic image. Therefore, byapplying the distance image information included in the distance image(virtual endoscopic image) to the endoscopic image, it is possible todetermine the distance information (distance image information in theendoscopic image) such as a distance between internal body partsincluded in the endoscopic image and a size of the internal body part.

The operation assistance information output unit 626 acquires thedistance image information which is output from the distance imageinformation calculation unit 625 and the viewpoint position and thedirection (rotation angle) of the endoscope 40. The operation assistanceinformation output unit 626 acquires the three-dimensional medical imageoutput from the acquisition unit 621, and extracts body cavityinformation included in the three-dimensional medical image. Asdescribed above, the body cavity information included in thethree-dimensional medical image is, for example, curved surface dataindicating a shape (a shape of an inner wall of an organ) of an internalorgan into which the endoscope 40 is inserted in a three-dimensionalregion including the imaging region of the virtual endoscopic image fromwhich the distance image information is calculated.

The operation assistance information output unit 626 includes anoperation information learning model 91 that outputs operationassistance information including an entry direction of the endoscope 40based on the input distance image information, the viewpoint positionand the direction of the endoscope 40, and the body cavity informationwhich is included in the three-dimensional medical image and isrepresented by curved surface data. The operation assistance informationoutput unit 626 inputs, to the operation information learning model 91,the acquired distance image information, the viewpoint position and thedirection of the endoscope 40, and the body cavity information which isincluded in the three-dimensional medical image and is represented bycurved surface data, and acquires the operation assistance informationwhich includes the entry direction of the endoscope 40 and is outputfrom the operation information learning model 91.

As described above, the operation assistance information which is outputfrom the operation information learning model 91 includes, for example,information on an insertion direction, an insertion amount, or aninsertion speed of the endoscope 40, from the viewpoint position and thedirection of the endoscope 40 at the present time, that is, at the timeof capturing the endoscopic image, to a target point indicating aninsertion destination. The operation assistance information may bedisplayed in a vector format or a matrix format in the coordinate system(internal body coordinate system) of the three-dimensional medicalimage, the format including a plurality of coordinate values and aplurality of rotation angles through which the distal end portion 443 ofthe endoscope 40 sequentially passes from the viewpoint position of theendoscope 40 to the target point. By using the plurality of coordinatevalues and the plurality of rotation angles through which the endoscope40 sequentially passes, it is possible to determine an insertion path ofthe endoscope 40 by connecting the plurality of coordinate values aspath points. When calculating each of the coordinate values as the pathpoints, the operation assistance information output unit 626 may performcorrection according to hardness of the insertion portion (flexibletube) 44 and calculate the coordinate values.

The operation assistance information output unit 626 acquires theendoscopic image output from the acquisition unit 621, generates, forexample, image data obtained by superimposing the operation assistanceinformation on the endoscopic image, and outputs the image data to thedisplay unit 7. The display unit 7 displays the endoscopic imageobtained by superimposing the operation assistance information on theendoscopic image, based on the image data acquired from the operationassistance information output unit 626.

The subject ID, the examination date and time, the endoscopic image, theS coordinate, and the three-dimensional medical image, which areacquired by the acquisition unit 621, the virtual endoscopic imagecalculated by the matching degree determination unit 624, and theinformation on the viewpoint position and the direction of the endoscope40 are stored in the endoscopic image DB in association with each other.That is, the control unit of the information processing apparatus mayfunction as a DB registration unit, and register and store variousimages, information, or data acquired or calculated by the acquisitionunit 621 and the matching degree determination unit 624 in theendoscopic image DB.

FIG. 9 is an explanatory diagram on a relationship between theendoscopic image and the three-dimensional medical image. In FIG. 9, arelationship between the three-dimensional medical image, the virtualendoscopic image, and the endoscopic image is represented in anobject-oriented manner.

As described above, the three-dimensional medical image, the virtualendoscopic image, and the endoscopic image which are registered in theendoscopic image DB 631 are associated with each other based on theviewpoint position and the viewpoint direction at the time of capturingthe endoscopic image. The viewpoint position corresponds to a coordinate(x, y, z) in the coordinate system (internal body coordinate system) ofthe three-dimensional medical image. The viewpoint direction correspondsto a rotation angle (Θx, Θy, Θz) in an x-axis, a y-axis, and a z-axis inthe coordinate system (internal body coordinate system) of thethree-dimensional medical image.

Each pixel of the endoscopic image corresponds to each pixel of thevirtual endoscopic image (the virtual endoscopic image having a highestmatching degree with the endoscopic image). The virtual endoscopic imageis an image generated by setting, as a starting point, the viewpointposition based on the three-dimensional medical image, and performingprojection by vector conversion using a viewpoint vector defined by theviewpoint direction (rotation angle). A coordinate in the coordinatesystem (internal body coordinate system) of the three-dimensionalmedical image is determined by a pixel of the virtual endoscopic image.

As described above, since each of the pixels of the virtual endoscopicimage corresponds to each of the pixels of the endoscopic image, it ispossible to determine the coordinate of the pixel of the endoscopicimage, that is, the coordinate of the internal body part included in theendoscopic image in the coordinate system (internal body coordinatesystem) of the three-dimensional medical image, based on the pixel ofthe virtual endoscopic image. That is, by using the virtual endoscopicimage as an intermediate medium, it is possible to associate the pixel(internal body part) of the endoscopic image with the coordinate in thecoordinate system (internal body coordinate system) of thethree-dimensional medical image.

Color information and narrow band pixel information of pixels of theendoscopic image may be added to the three-dimensional medical image,and the three-dimensional medical image may be registered in theendoscopic image DB 631. In a case where the pixel information of theendoscopic image, such as the difference and the color information, isadded to the three-dimensional medical image, it is desirable to performluminance correction by using an imaging light source 446. As describedabove, the distance between the pixel of the endoscopic image and theviewpoint position (a position of the imaging light source 446) iscalculated on the coordinate system of the three-dimensional medicalimage. Therefore, brightness included in the pixel information of theendoscopic image may be corrected based on a reciprocal obtained bysquaring the calculated distance. In a case where there are a pluralityof endoscopic images including pixels located at the same coordinates inthe coordinate system of the three-dimensional medical image, the pixelinformation may be added to the three-dimensional medical image byapplying a weight according to the distance, such as giving preferenceto the endoscopic image having a shortest distance, and performing loadaveraging or simple averaging.

In imaging of a three-dimensional medical image, in a case where anX-ray CT apparatus or a cone beam CT apparatus is used, for example,dual energy imaging may be performed, and an image in which acomposition (body composition) of each pixel of a three-dimensionalmedical image can be identified by the effective mass number(effective-Z) may be used. Further, in a case where an MRI apparatus isused, an image obtained by adding information on a composition (bodycomposition) of each pixel of a three-dimensional medical image, such asfat or lactic acid, may be used. As described above, by addinginformation on the body composition, such as the effective mass number(effective-Z), fat, or lactic acid, to the composition of each pixel ofthe three-dimensional medical image, it is possible to provide, for adoctor or the like, information in which the added information and theendoscopic image associated with the coordinate specified by each pixelof the three-dimensional medical image are associated with each other.

FIG. 10 is a flowchart illustrating an example of a processing procedureperformed by the control unit 62 of the information processing apparatus6. For example, the information processing apparatus 6 starts processingof the flowchart based on contents input from the input unit 8 connectedto the information processing apparatus 6.

The control unit 62 of the information processing apparatus 6 acquiresthe examination date and time, the subject ID, the endoscopic image, andthe information on the insertion distance which are output from theendoscope processor 20 (step S101). The endoscopic image acquired by thecontrol unit 62 from the endoscope processor 20 may be a still image ora moving image. In addition to acquisition of the endoscopic image, thecontrol unit 62 acquires information on the insertion distance of theendoscope 40 which is output from the optical sensor or the like, theexamination dale and time (imaging date and time of the endoscopicimage), and attribute information of the subject such as the subject ID.

The control unit 62 of the information processing apparatus 6 acquires athree-dimensional medical image which is output from an examinationapparatus configured to capture a three-dimensional image of theinternal body, such as a CT apparatus, an MRI apparatus, or anultrasonic diagnosis apparatus (step S102). The three-dimensionalmedical image may be acquired by the information processing apparatus 6communicatively connected to an examination apparatus configured tocapture a three-dimensional image of the internal body, such as a CTapparatus, an MRI apparatus, or an ultrasonic diagnosis apparatus.Alternatively, in a case where a three-dimensional medical image, whichis output front an examination apparatus configured to capture athree-dimensional image of the internal body, such as a CT apparatus, anMRI apparatus, or an ultrasonic diagnosis apparatus, is already storedin, for example, an external server (not illustrated), the informationprocessing apparatus 6 may access the external server, and acquire thethree-dimensional medical image of the subject based on the subject IDoutput from the endoscope processor 20. Alternatively, in a case wherethe endoscope processor 20 is communicatively connected to anexamination apparatus configured to capture a three-dimensional image ofthe internal body, such as a CT apparatus, an MRI apparatus, or anultrasonic diagnosis apparatus, the control unit 62 of the informationprocessing apparatus 6 may acquire a three-dimensional medical imagefrom the examination apparatus configured to capture a three-dimensionalimage of the internal body, such as a CT apparatus, an MRI apparatus, oran ultrasonic diagnosis apparatus, via the endoscope processor 20.

The control unit 62 of the information processing apparatus 6 calculatesthe viewpoint position based on the insertion distance (S coordinate)(step S103). The control unit 62 acquires information on the insertiondistance (S coordinate) from, for example, an optical sensor or the likedisposed inside the insertion portion 44 (flexible tube) of theendoscope 40 via the endoscope processor 20, and calculates a coordinateof the distal end portion 443 of the endoscope 40 located inside theinner wall of the digestive organ into which the endoscope 40 isinserted based on the acquired insertion distance (S coordinate) and thethree-dimensional medical image. The coordinate is a coordinate in thecoordinate system (internal body coordinate system) of thethree-dimensional medical image when a predetermined point is set as anorigin.

The control unit 62 of the information processing apparatus 6 generatesa plurality of candidate virtual endoscopic images based on theviewpoint position (step S104). The control unit 62 sequentiallygenerates a plurality of candidate virtual endoscopic images by changingthe viewpoint direction, that is, the rotation angle (Θx, Θy, Θz) in thecoordinate system of the three-dimensional medical image by apredetermined unit amount, from the viewpoint position corresponding tothe coordinate of the distal end portion 443 of the endoscope 40 as astarting point. For example, in a case where the predetermined unitamount is 10°, the control unit 62 has 36 resolutions for the rotationangle of each axis. That is, the control unit 62 may generate 46656 (36to the power of 3) candidate virtual endoscopic images.

The control unit 62 of the information processing apparatus 6 specifiesa virtual endoscopic image having a highest matching degree with theendoscopic image among the plurality of generated virtual endoscopicimages (step S105). The control unit 62 specifies a virtual endoscopicimage having a highest matching degree with the endoscopic image, forexample, by using the matching degree learning model 92. Alternatively,the control unit 62 may measure the matching degree by using an indexindicating a correlation between a shade image of the endoscopic imageand a shade image of the virtual endoscopic image. The control unit 62specifies a virtual endoscopic image having a highest matching degreeand the viewpoint position and the direction (rotation angle) at thetime of generating the virtual endoscopic image.

The control unit 62 of the information processing apparatus 6 calculatesdistance image information from the acquired virtual endoscopic image(step S106). The control unit 62 calculates distance image informationthat is information on the distance between pixels in the virtualendoscopic image.

The control unit 62 of the information processing apparatus 6 outputsoperation assistance information based on the distance image informationand the body cavity information included in the three-dimensionalmedical image (step S107). The control unit 62 extracts, the body cavityinformation included in the acquired three-dimensional medical image, ascurved surface data indicating a shape (a shape of an inner wall of anorgan) of an internal organ into which the endoscope 40 is inserted in athree-dimensional region including the imaging region of the virtualendoscopic image from which the distance image information iscalculated. The control unit 62 inputs, to the operation informationlearning model 91, the extracted curved surface data, the distance imageinformation, and information indicating the position and the directionof the endoscope 40, and outputs the operation assistance informationoutput by the operation information learning model 91.

When outputting the operation assistance information, the control unit62 may generate image data obtained by superimposing the operationassistance information on the endoscopic image, the virtual endoscopicimage, or the three-dimensional medical image, and output the image datato, for example, the display unit. The display unit displays theendoscopic image, the virtual endoscopic image, or the three-dimensionalmedical image on which the operation assistance information issuperimposed, based on the image data output from the control unit 62 ofthe information processing apparatus 6.

The operation assistance information superimposed on the endoscopicimage, the virtual endoscopic image, or the three-dimensional medicalimage includes the insertion direction, the insertion amount, and theinsertion speed of the endoscope 40, and information on the target pointcoordinate indicating the insertion destination. Thus, usefulinformation can be provided for an operator of the endoscope 40 such asa doctor, and it is possible to contribute to diagnosis assistance for adoctor and the like.

FIG. 11 is an explanatory diagram illustrating an aspect of anintegrated image display screen 71. As described above, the control unit62 of the information processing apparatus 6 generates image dataincluding the endoscopic image, the virtual endoscopic image, or thethree-dimensional medical image on which the operation assistanceinformation such as the insertion direction of the endoscope 40 issuperimposed, and outputs the image data to the display unit 7. Theintegrated image display screen 71 is an example of a display screenconfigured by the image data, and the display unit 7 displays theintegrated image display screen 71 based on the image data.

The integrated image display screen 71 includes, for example, a regionfor displaying bibliographic mailers such as a subject ID, a region fordisplaying an endoscopic image, a region for displaying athree-dimensional medical image, a region for displaying atwo-dimensional medical image, a region for displaying a virtualendoscopic image, and a region for displaying information on thecurrently-displayed endoscopic image, a viewpoint position from whichthe endoscopic image is captured, and the like.

In the region for displaying bibliographic matters such as a subject ID,bibliographic matters in data management, such as the subject ID used tospecify the three-dimensional medical image corresponding to theendoscopic image, the examination date and time by the endoscope 40, anda generation date of the three-dimensional medical image, are displayed.

In the region for displaying the endoscopic image, the endoscopic imagecurrently captured by the endoscope 40 is displayed in real time. Theoperation assistance information such as the insertion direction of theendoscope 40 is displayed by being superimposed on the endoscopic image.In accordance with setting of a display option to be described later,the endoscopic image may be displayed in a translucent manner, and theinternal body part located on an inner side of the internal wall surfacedisplayed in the endoscopic image may be displayed by a dotted line, theinternal body part being extracted from the virtual endoscopic imagecorresponding to the endoscopic image. The internal body part displayedby a dotted line in translucent display may be, for example, a part of alesion candidate extracted based on shape information of the internalbody part specified in the three-dimensional medical image.

In the region for displaying the three-dimensional medical image, theinternal body part such as a digestive organ represented in thethree-dimensional medical image is displayed as a three-dimensionalobject, and the operation assistance information such as the viewpointposition of the endoscope 40 and the insertion direction of theendoscope 40 when the viewpoint position is set as a starting point isdisplayed by being superimposed. By dragging a part of thethree-dimensional object, the three-dimensional object can be rotated.In the three-dimensional medical image, a position of a part of a lesioncandidate extracted based on shape information of an internal body partspecified in the three-dimensional medical image may he displayed, forexample, in a highlighted state.

In the region for displaying the two-dimensional medical image, atwo-dimensional medical image obtained by projecting, on thethree-dimensional medical image, a region where the operation assistanceinformation such as the insertion direction of the endoscope 40 issuperimposed is displayed. A projection vector used for generating thetwo-dimensional medical image may be determined in accordance with arotation state of the three-dimensional object displayed in the regionwhere the three-dimensional medical image is displayed.

In the region for displaying the virtual endoscopic image, a virtualendoscopic image having a highest matching degree with the endoscopicimage is displayed, the endoscopic image being displayed in the regionfor displaying the endoscopic image. The operation assistanceinformation such as the insertion direction of the endoscope 40 may bedisplayed by being superimposed on the virtual endoscopic image as inthe endoscopic image.

In the region for displaying the viewpoint position when the endoscopicimage is captured, the position (viewpoint position) and the viewpointdirection (rotation angle) of the endoscope 40 in the body at the timeof capturing the endoscopic image are displayed, the endoscopic imagebeing displayed in the region for displaying the endoscopic image. Asdescribed above, the control unit 62 (acquisition unit 621) of theinformation processing apparatus 6 continuously acquires the endoscopicimage and the S coordinate indicating the insertion distance of theendoscope 40 from the processor for the endoscope 40, and continuouslycalculates the position (viewpoint position) of the distal end portionof the endoscope 40 based on the acquired S coordinate. Further, thecontrol unit 62 (acquisition unit 621) of the information processingapparatus 6 continuously calculates the direction (rotation angle) ofthe distal end portion of the endoscope 40 when specifying the virtualendoscopic image corresponding to the endoscopic image based on thematching degree with the endoscopic image. Therefore, in the region fordisplaying the viewpoint position when the endoscopic image is captured,the viewpoint position and the viewpoint direction of the endoscope 40are displayed in real time according to an operation of the endoscope 40by a doctor or the like.

In the region for displaying information of the currently-displayedendoscopic image, for example, information on an internal body part or apixel at the image center of the currently-displayed endoscopic image isdisplayed. As described above, in the three-dimensional medical image,the information on the internal body part (pixel) includes informationon the composition (body composition) of each pixel of thethree-dimensional medical image, such as the effective mass number(effective-Z), fat, or lactic acid which is substance determinationinformation based on X-rays. Therefore, it is possible to display theinformation on the body composition such as the effective mass number(effective-Z) extracted from the three-dimensional medical image, in theregion, based on the coordinate in the internal body coordinate systemindicating the image center of the endoscopic image. Further, thepresence or absence of a lesion in an internal body part included in thecurrently-displayed endoscopic image can also be displayed in the regionby using a learning model that receives the endoscopic image and outputsinformation on the presence or absence of a lesion. As the learningmodel that outputs the information on the presence or absence of alesion based on the input endoscopic image, for example, any objectdetection algorithm having a function of a segmentation network such asCNN, regions with convolutional neural network (RCNN), Fast RCNN,Faster-RCNN, single shot multibox detector (SSD), or You Only Look Once(YOLO) may be used.

As described above, since the viewpoint position of the endoscope 40 iscontinuously calculated according to the operation of the endoscope 40by a doctor or the like, for example, the information on the distancebetween the viewpoint position and the image center of thecurrently-displayed endoscopic image may be calculated, and thecalculated distance (distance from the viewpoint position to the pixelat the image center) may be displayed in the region.

The integrated image display screen 71 includes an input region forreceiving an input on a display mode, and in the input region, forexample, a display option field 711 for setting a display option and adisplay mode selection field 712 for receiving selection of a pluralityof display modes are disposed.

The display option field 711 is provided with a toggle switch forsetting whether to translucently display the endoscopic image, thevirtual endoscopic image, or both images. In a case where a check isinput in the toggle switch, based on shape data of the internal bodypart included in the virtual endoscopic image or the three-dimensionalmedical image, processing of making the internal wall surface displayedin the endoscopic image translucent is performed, and the internal bodypart located on an inner side of the internal wall surface is displayedby a dotted line. As described above, the internal body part located onan inner side of the internal wall surface may be, for example, a partof a lesion candidate extracted based on shape information of theinternal body part specified in the three-dimensional medical image.

The display mode selection field 712 is provided with toggle switchesfor selecting the virtual endoscopic image, the two-dimensional medicalimage, and the three-dimensional medical image which are to be displayedtogether with the endoscopic image. In a case where a check is input inthe toggle switch corresponding to any one of these images, the imagecorresponding to the check is displayed. According to the check which isinput in each of the toggle switches in the display mode selection field712, it is possible to select a case of displaying any one or two imagesof the virtual endoscopic image, the two-dimensional medical image, andthe three-dimensional medical image and a case of displaying all ofthese three images. A display size of the image may be resized accordingto the number of images to be displayed.

According to the present embodiment, based on the endoscopic image, athree-dimensional medical image is acquired by imaging an internal bodyportion of the subject by, for example, an examination apparatusconfigured to capture a three-dimensional image of an internal bodyportion of the subject, such as X-ray CT, X-ray cone beam CT, MRI-CT, oran ultrasonic diagnosis apparatus, the virtual endoscopic image isreconfigured from the acquired three-dimensional medical image, anddistance image information in the endoscopic image is calculated basedon the virtual endoscopic image and the endoscopic image. Therefore, byusing, in addition to the endoscopic image, the virtual endoscopic imageconfigured with the three-dimensional medical image including coordinateinformation in a three-dimensional space, it is possible to accuratelycalculate distance image information in the endoscopic image. Based onthe calculated distance image information and the three-dimensionalmedical image or the virtual endoscopic image, it is possible toefficiently output the operation assistance information on the operationof the endoscope. The X-ray CT, the X-ray cone beam CT, the MRI-CT, orthe ultrasonic diagnosis apparatus is an example of an examinationapparatus configured to capture a three-dimensional image of an internalportion of a body, and is not limited thereto. The three-dimensionalmedical image is not limited to a case of being acquired from any ofthese examination apparatuses, and may be acquired from a plurality ofexamination apparatuses.

According to the present embodiment, the operation assistanceinformation includes the information on the insertion direction or theinsertion speed of the endoscope 40. Thus, by outputting the operationassistance information, for an operator of the endoscope 40 such as adoctor, it is possible to efficiently perform diagnosis assistance on anoperation of the endoscope 40.

According to the present embodiment, the operation assistanceinformation is displayed, for example, on the integrated image displayscreen 71 by being superimposed on the currently-captured endoscopicimage, and the three-dimensional medical image, the two-dimensionalmedical image, or the virtual endoscopic image, which corresponds to theendoscopic image. Therefore, visibility of the operation assistanceinformation for an operator of the endoscope 40 such as a doctor can beimproved, and thus it is possible to efficiently perform diagnosisassistance for a doctor or the like.

Second Embodiment

The information processing apparatus 6 according to a second embodimentis different from the information processing apparatus according to thefirst embodiment in that the viewpoint position is corrected based on abending history acquired from the endoscope processor 20. FIG. 12 is afunctional block diagram exemplifying functional units included in thecontrol unit of the information processing apparatus according to asecond embodiment (bending history).

The control unit 21 of the endoscope processor 20 acquires bendinghistory information of the endoscope 40 inserted into the body, anddetermines an insertion situation of the endoscope 40 according to theacquired bending history information. The control unit 21 of theendoscope processor 20 ma detect the bending history information byusing, for example, an endoscope-insertion-shape detection device (notillustrated) connected to the endoscope processor 20. For example, asdisclosed in JP 2019-37643 A, the endoscope-insertion-shape detectiondevice may be a device in which a plurality of magnetic coils aredisposed inside the insertion portion 44 of the endoscope 40 atpredetermined intervals along a longitudinal direction of the insertionportion 44. The bending history information indicates a physicalparameter or information on bending such as a bending angle or a bendingdirection.

As in the first embodiment, the acquisition unit 621 of the informationprocessing apparatus 6 acquires an endoscopic image from the endoscopeprocessor 20, and further acquires bending history information. Theacquisition unit 621 outputs the acquired bending history information tothe viewpoint position calculation unit 622.

The viewpoint position calculation unit 622 corrects the insertiondistance (S coordinate) based on the acquired bending historyinformation, and calculates the viewpoint position based on thecorrected insertion distance (S coordinate) as in the first embodiment.The viewpoint position calculation unit 622 detects a shape (forexample, rightward bending by 30 degrees) of the insertion portion 44 byarithmetic processing according to the bending angle and the bendingdirection. The control unit 21 recalculates (corrects) the S coordinatethat is the insertion distance based on the detected shape of theinsertion portion 44. Thereafter, each functional unit such as thevirtual endoscopic image generation unit 623 performs processing as inthe first embodiment, and the operation assistance information outputunit 626 generates and outputs operation assistance information as inthe first embodiment.

The viewpoint position calculation unit 622 of the informationprocessing apparatus 6 corrects the viewpoint position based on thebending history acquired from the endoscope processor 20, and thepresent invention is not limited thereto. The control unit 21 of theendoscope processor 20 may correct the insertion distance based on theacquired bending history information, and output the corrected insertiondistance to the information processing apparatus 6. The acquisition unit621 of the information processing apparatus 6 may acquire the viewpointposition which is corrected based on the bending history information bythe control unit 21 of the endoscope processor 20, and the subsequentprocessing may be performed as in the first embodiment.

The position information for associating the endoscopic image with thethree-dimensional medical image is calculated based on the informationon the bending history, the information on the insertion distance, andthe length of the insertion path of the endoscope 40 that is specifiedin the three-dimensional medical image. By correcting the information onthe insertion distance based on the information on the bending history,accuracy of the insertion distance (S coordinate) can be improved.Therefore, it is possible to accurately specify the viewpoint position(coordinate) and the viewpoint direction (rotation angle) of theendoscope 40 in the coordinate system of the three-dimensional medicalimage at the time of capturing the endoscopic image. Thus, it ispossible to efficiently generate a suitable virtual endoscopic image.Thereby, it is possible to further improve accuracy in associationbetween the endoscopic image and the three-dimensional medical image.

FIG. 13 is a flowchart illustrating an example of a processing procedureperformed by the control unit of the information processing apparatus.For example, the information processing apparatus 6 starts processing ofthe flowchart based on contents input from the input unit 8 connected tothe information processing apparatus 6.

The control unit 62 of the information processing apparatus 6 acquiresthe examination date and time, the subject ID, the endoscopic image, theinsertion distance, and the information on the bending history which areoutput from the endoscope processor 20 (step S201). The control unit 62of the information processing apparatus 6 acquires the endoscopic imagefrom the endoscope processor 20 as in the first embodiment, and furtheracquires, for example, information on the bending history detected by anendoscope-insertion-shape observation device via the endoscope processor20. Alternatively, the control unit 62 of the information processingapparatus 6 may directly acquire the information on the bending historyfrom the endoscope-insertion-shape observation device.

The control unit 62 of the information processing apparatus 6 acquires athree-dimensional medical image which is output from an examinationapparatus configured. to capture a three-dimensional image of theinternal body, such as a CT apparatus, an MRI apparatus, or anultrasonic diagnosis apparatus (step S202). The control unit 62 of theinformation processing apparatus 6 performs processing of step S202similarly to processing of step S102 of the first embodiment.

The control unit 62 of the information processing apparatus 6 correctsthe insertion distance (S coordinate) based on the bending history whichis output from the endoscope processor 20, and calculates the viewpointposition (step S203). The control unit 62 calculates a shape of theinsertion portion 44 (for example, rightward bending by 30 degrees) byarithmetic processing according to the bending angle and the bendingdirection included in the bending history, and recalculates (corrects)the S coordinate that is the insertion distance based on the calculatedshape of the insertion portion 44. The control unit 62 calculates theviewpoint position based on the corrected insertion distance (Scoordinate) as in the first embodiment.

The control unit 62 of the information processing apparatus 6 generatesa plurality of candidate virtual endoscopic images based on theviewpoint position (step S204). The control unit 62 of the informationprocessing apparatus 6 specifies a virtual endoscopic image having ahighest matching degree with the endoscopic image among the plurality ofgenerated virtual endoscopic images (step S205). The control unit 62 ofthe information processing apparatus 6 calculates distance imageinformation from the acquired virtual endoscopic image (step S206). Thecontrol unit 62 of the information processing apparatus 6 outputsoperation assistance information based on the distance image informationand the body cavity information included in the three-dimensionalmedical image (step S207). The control unit 62 of the informationprocessing apparatus 6 performs processing of step 204, step S205, stepS206, and step S207 similarly to processing of step S104, step S105,step S106, and step S107 of the first embodiment.

According to the present embodiment, the coordinate information of theendoscope 40 and the information (viewpoint position) on the directionof the endoscope 40 in the coordinate system of the three-dimensionalmedical image are calculated based on the information on the shape ofthe endoscope 40, the information on the bending history, theinformation on the insertion distance, and the three-dimensional medicalimage. Therefore, it is possible to specify the position and therotation angle (rotation angle in the x-axis, the y-axis, and thez-axis) of the endoscope 40 in the coordinate system of thethree-dimensional medical image at the time of capturing the endoscopicimage according to the shape of the endoscope 40. Thus, it is possibleto accurately and efficiently calculate the distance image informationin the endoscopic image based on the position and the rotation angle ofthe endoscope 40.

Third Embodiment

FIG. 14 is a schematic diagram illustrating an outline of the diagnosisassistance system S according to a third embodiment (automatic operationmechanism 434). FIG. 15 is a functional block diagram exemplifyingfunctional units included in the control unit 62 of the informationprocessing apparatus 6. The diagnosis assistance system S according tothe third embodiment is different from the diagnosis assistance systemaccording to the first embodiment in that the endoscope 40 included inthe diagnosis assistance system S further includes an automaticoperation mechanism 434.

The endoscope 40 included in the diagnosis assistance system S includesan automatic operation mechanism 434 that automatically operates thecontrol button 431 and the bending knob 433 based on the operationassistance information output from the control unit 62 (operationassistance information output unit 626) of the information processingapparatus 6.

The automatic operation mechanism 434 is communicatively connected tothe information processing apparatus 6, and acquires (receives) theoperation assistance information which is output (transmitted) from theinformation processing apparatus 6. The automatic operation mechanism434 includes, for example, a microcomputer (not illustrated) thatgenerates an on/off signal or a pulse signal for the control button 431or the bending knob 433 from the acquired operation assistanceinformation, and a motor/cam mechanism (not illustrated) that operatesor drives the control button 431 or the bending knob 433 based on thesignal output from the microcomputer. In this manner, the automaticoperation mechanism 434, the control button 431, and the bending knob433 cooperate with each other based on the operation assistanceinformation which is output from the information processing apparatus 6,and perform automatic operation such as automatic insertion of theendoscope 40 into the body of the subject according to the operationassistance information.

The operation assistance information that is acquired by the automaticoperation mechanism 434 from the control unit 62 (operation assistanceinformation output unit 626) of the information processing apparatus 6is not limited to the operation assistance information such as insertionand bending of the insertion portion 44. For example, in a case wherethe insertion portion 44 of the endoscope 40 is provided with an airinjection portion (not illustrated) or a hand portion (not illustrated),the operation assistance information may include information on anoperation such as injection of air by using the air injection portion orextraction (sampling) of a lesion part by using the hand portion. Thatis, the operation assistance information output unit 626 generatesinformation on an operation of the air injection portion or the handportion based on the shape information and the distance imageinformation of the internal body part specified in the acquiredthree-dimensional medical image, includes the information in theoperation assistance information, and outputs the operation assistanceinformation to the automatic operation mechanism 434. The automaticoperation mechanism 434 may automatically operate the air injectionportion or the hand portion based on the acquired operation assistanceinformation. The information on the operation of the air injectionportion or the hand portion included in the operation assistanceinformation may be displayed on the integrated image display screen 71by being superimposed on the endoscopic image.

According to the present embodiment, based on the distance imageinformation in the endoscopic image, and the three-dimensional medicalimage or the virtual endoscopic image, it is possible to efficientlyoutput the operation assistance information on the operation of theendoscope 40. Thus, the automatic operation mechanism 434 automaticallyoperates the endoscope 40 according to the operation assistanceinformation which is output from the endoscope processor 20. Therefore,it is possible to provide, for an operator who operates the endoscope 40such as a doctor, the diagnosis assistance system S that efficientlyassists an operation of the endoscope 40.

The embodiments herein are disclosed for purposes of illustration in allrespects without being limited. The technical features described in therespective embodiments can be combined with each other, and the scope ofthe present invention is intended to include all modifications withinthe scope of the claims and the scope equivalent to the claims.

REFERENCE SIGNS LIST

-   S Diagnosis assistance system-   10 Endoscope apparatus-   15 Keyboard-   16 Storage shelf-   20 Endoscope processor-   21 Control unit-   211 Image processing unit-   22 Main storage device-   23 Auxiliary storage device-   24 Communication unit-   25 Touch panel-   26 Display device IN-   27 Input device OF-   28 Reading unit-   31 Endoscope connector-   311 Electrical connector-   312 Optical connector-   33 Light source-   34 Pump-   35 Water supply tank-   36 Air supply/water supply port-   40 Endoscope-   43 Operation unit-   431 Control button-   433 Bending knob-   434 Automatic operation mechanism-   44 Insertion portion (flexible tube)-   441 Soft portion-   442 Bending portion-   443 Distal end portion-   444 Imaging unit-   446 Imaging light source-   45 Bend preventing portion-   48 Scope connector-   49 Universal cord-   50 Display device-   6 Information processing apparatus-   61 Communication unit-   62 Control unit-   621 Acquisition unit-   622 Viewpoint position calculation unit-   623 Virtual endoscopic image generation unit-   624 Matching degree determination unit-   625 Distance image information calculation unit-   626 Operation assistance information output unit-   63 Storage unit-   631 Endoscopic image DB-   632 Recording medium-   P Program-   64 Input/output I/F-   7 Display unit-   71 Integrated image display screen.-   711 Display option field-   712 Display mode selection field-   8 Input unit-   91 Operation information learning model-   92 Matching degree learning model

1. A non-transitory computer-readable medium containing a programcausing a computer to execute processing comprising: acquiring anendoscopic image of a subject from an endoscope; acquiring athree-dimensional medical image obtained by capturing an image of aninternal body portion of the subject by means of X-ray CT, X-ray conebeam CT, MRI-CT, or an ultrasonic diagnosis apparatus configured tocapture a three-dimensional image of an internal body portion of thesubject; generating a virtual endoscopic image reconfigured from thethree-dimensional medical image, based on the acquired endoscopic image;calculating distance image information in the endoscopic image based onthe virtual endoscopic image and the endoscopic image; and outputtingoperation assistance information on an operation of the endoscope basedon the distance image information and the three-dimensional medicalimage.
 2. The non-transitory computer-readable medium containing aprogram according to claim 1, wherein the operation assistanceinformation including information on an insertion direction or aninsertion speed of the endoscope and an insertion distance of theendoscope is output based on the distance image information and thethree-dimensional medical image.
 3. The non-transitory computer-readablemedium containing a program according to claim 1, wherein the distanceimage information including information on a size of an internal bodypart included in the endoscopic image or a distance between the internalbody parts is calculated based on the virtual endoscopic image and theendoscopic image.
 4. The non-transitory computer-readable mediumcontaining a program according to claim 1, wherein information on aninsertion distance of the endoscope inserted into the internal bodyportion of the subject is acquired, coordinate information of theendoscope in a coordinate system of the three-dimensional medical imageis calculated based on the information on the insertion distance and thethree-dimensional medical image, and the operation assistanceinformation on an operation of the endoscope is output based on thecoordinate information of the endoscope, the distance image information,and the three-dimensional medical image.
 5. The non-transitorycomputer-readable medium containing a program according to claim 4,wherein information on a bending history of the endoscope inserted intothe internal body portion of the subject is acquired, coordinateinformation of the endoscope and information on a direction of theendoscope in a coordinate system of the three-dimensional medical imageare calculated based on the information on the bending history, theinformation on the insertion distance, and the three-dimensional medicalimage, and the operation assistance information on an operation of theendoscope is output based on the coordinate information of theendoscope, the distance image information, and the three-dimensionalmedical image.
 6. The non-transitory computer-readable medium containinga program according to claim 5, wherein information on a shape of theendoscope is acquired, coordinate information of the endoscope andinformation on a direction of the endoscope in a coordinate system ofthe three-dimensional medical image are calculated based on theinformation on the shape of the endoscope, the information on thebending history, the information on the insertion distance, and thethree-dimensional medical image, and the operation assistanceinformation on an operation of the endoscope is output based on thecoordinate information of the endoscope, the information on thedirection of the endoscope, the virtual endoscopic image, and theendoscopic image.
 7. The non-transitory computer-readable mediumcontaining a program according to claim 1, wherein the generating of thevirtual endoscopic image includes processing of generating atwo-dimensional medical image obtained by projecting thethree-dimensional medical image, and setting, in the generatedtwo-dimensional medical image, as the virtual endoscopic image, atwo-dimensional medical image in which a difference from the endoscopicimage is equal to or smaller than a predetermined value.
 8. Thenon-transitory computer-readable medium containing a program accordingto claim 1, wherein the generating of the virtual endoscopic imageincludes processing of generating a two-dimensional medical imageobtained by projecting the three-dimensional medical image, inputtingthe two-dimensional medical image and the endoscopic image to a learningmodel, which is learned so as to output a matching degree between thetwo-dimensional medical image and the endoscopic image, in a case wherethe two-dimensional medical image and the endoscopic image are input,acquiring a matching degree between the two-dimensional medical imageand the endoscopic image from the learning model, and setting, as thevirtual endoscopic image, the two-dimensional medical image in which thematching degree between the two-dimensional medical image and theendoscopic image is equal to or higher than a predetermined value.
 9. Aninformation processing method causing a computer to execute processingcomprising: acquiring an endoscopic image of a subject from anendoscope; acquiring a three-dimensional medical image obtained bycapturing an image of an internal body portion of the subject by meansof X-ray CT, X-ray cone beam CT, MRI-CT, or an ultrasonic diagnosisapparatus configured to capture a three-dimensional image of an internalbody portion of the subject; generating a virtual endoscopic imagereconfigured from the three-dimensional medical image, based on theacquired endoscopic image; calculating distance image information in theendoscopic image based on the virtual endoscopic image and theendoscopic image; and outputting operation assistance information on anoperation of the endoscope based on the distance image information andthe three-dimensional medical image.
 10. An information processingapparatus comprising: an endoscopic image acquisition unit that acquiresan endoscopic image of a subject from an endoscope; a three-dimensionalmedical image acquisition unit that acquires a three-dimensional medicalimage obtained by capturing an image of an internal body portion of thesubject by means of X-ray CT, X-ray cone beam CT, MRI-CT, or anultrasonic diagnosis apparatus configured to capture a three-dimensionalimage of an internal body portion of the subject; a generation unit thatgenerates a virtual endoscopic image reconfigured from thethree-dimensional medical image, based on the acquired endoscopic image;a calculation unit that calculates distance image information in theendoscopic image based on the virtual endoscopic image and theendoscopic image; and an output unit that outputs operation assistanceinformation on an operation of the endoscope based on the distance imageinformation and the three-dimensional medical image.
 11. (canceled)